System and method for 3-D digital reconstruction of an oral cavity from a sequence of 2-D images

ABSTRACT

Systems and methods are provided through which a model-based vision system for dentistry which assists in diagnosis, treatment planning and surgical simulation. The present invention includes an integrated computer vision system that constructs a three-dimensional (3-D) model of the patient&#39;s dental occlusion using an intra-oral video camera. A modified shape from shading technique, using perspective projection and camera calibration, extracts the 3-D information from a sequence of two-dimensional images of the jaw. Data fusion of range data and 3-D registration techniques develop a complete 3-D digital jaw model. Triangulation of the 3-D digital model is then performed, and optionally, a solid 3-D model is reconstructed.

RELATED APPLICATION

[0001] This application claims the benefit of U.S. ProvisionalApplication Serial No. 60/199,913, filed Apr. 26, 2000 under 35 U.S.C.119(e).

FIELD OF THE INVENTION

[0002] This invention relates generally to digital dental imaging, andmore particularly to three-dimensional dental imaging.

BACKGROUND OF THE INVENTION

[0003] Dentistry requires accurate 3-D representation of the teeth andjaw for diagnostic and treatment purposes. For example, orthodontictreatment involves the application, over time, of force systems to teethto correct malocclusion. In order to evaluate tooth movement progress,the orthodontist monitors this movement by means of visual inspection,intra-oral measurements, fabrication of casts, photographs, andradiographs; this process is both costly and time consuming. Moreover,repeated acquisition of radiographs may result in untoward effects.Obtaining a cast of the jaw is a complex operation for the dentist, anunpleasant experience for the patient, and also may not provide all thenecessary details of the jaw.

[0004] Oral and maxillofacial radiology provides the dentist withabundant 3-D information of the jaw. Current and evolving methodsinclude computed tomography (CT), tomosynthesis, tuned-aperture CT(TACT), and localized, or “cone-beam,” computed tomography. While oraland maxillofacial radiology is now widely accepted as a routinetechnique for dental examinations, the equipment is rather expensive andthe resolution is frequently too low for 3-D modeling of dentalstructures. Furthermore, the radiation dose required to enhance bothcontrast and spatial resolution can be unacceptably high.

[0005] Much effort has been focused recently on computerized diagnosisin dentistry. One solution is an expert system where cephalometricmeasurements are acquired manually from the analysis of radiographs andplaster models. Another solution provides a computer-vision techniquefor the acquisition of jaw data from inexpensive dental wafers, which iscapable of obtaining imprints of the teeth. Conventional 3-D systems fordental applications commonly rely on obtaining an intermediate solidmodel of the jaw (cast or teeth imprints) and then capturing the 3-Dinformation from that model. User interaction is needed in such systemsto determine the 3-D coordinates of fiducial reference points on adental cast. Other systems that measure the 3-D coordinates have beendeveloped using either mechanical contact or a traveling lightprinciple. Yet another conventional solution includes a range scannerbased on white light to reconstruct the cast. The scanner used thesubtractive light principle to create very thin shadow profiles on thecast.

[0006] For the reasons stated above, and for other reasons stated belowwhich will become apparent to those skilled in the art upon reading andunderstanding the present specification, there is a general need toreplace conventional approaches in diagnosis, treatment planning,surgical simulation and prosthetic replacements. More specifically,there is a need in the art for three-dimensional (3-D) dental imagerynot using expensive, low-resolution and potentially harmful radiography,intermediate physical casts. There is also a need for fabricating dentalcasts in a manner that does not require time consuming and non-renewabledirect application of material to the dental surfaces. Moreover, thereis a need for a data acquisition system that obtains sequences ofcalibrated video images, with respect to a common reference in 3-Dspace, of the upper and/or lower jaw using an intraoral cavity camera.There is also a need for methods of accurate 3-D reconstruction of theupper and/or lower jaw from the acquired sequence of intraoral cavityimages. There is a further need for a shape-from-shading process thatincorporates the parameters of the intraoral cavity camera. There is yetanother need for a robust process for the fusion of data acquired frommultiple views of the intraoral cavity camera. There is still anotherneed for the implementation of a fast an accurate 3-D registration.There is still yet another need for specific object segmentation andrecognition of individual tooth information for further analysis andsimulations. There is still yet a further need to enable study andsimulation of tooth movement based on finite element and deformablemodel methods.

SUMMARY OF THE INVENTION

[0007] The above-mentioned shortcomings, disadvantages and problems areaddressed by the present invention, which will be understood by readingand studying the following specification.

[0008] The present invention provides a system for dentistry andorthodontics to replace traditional approaches in diagnosis, treatmentplanning, surgical simulation, and prosthetic replacements. The presentinvention provides a data acquisition system that obtains sequences ofcalibrated video images, with respect to a common reference inthree-dimensional (3-D) space, of an upper/lower jaw using a smallintra-oral camera. The present invention also provides methods for 3-Dreconstruction from the acquired sequence of intra-oral images. Thepresent invention further provides an algorithm for shape from shadingthat incorporates camera parameters. The present invention additionallyprovides a process for the fusion of data acquired from multiple views,including the implementation of an accurate and fast 3-D dataregistration. The present invention also provides an object segmentationand recognition system to separate and recognize individual 3-D toothinformation for further analysis and simulations. The present inventionin addition provides methods to enable study and simulation of toothmovement based on the finite element method and deformable modelapproaches. In varying embodiment, the present invention is implementedin various dental practices including implants, tooth alignment,craniofacial surgery, teledentistry, dental education and training, andthe analysis and simulation of dental operations including toothalignment, implant planing, restoration, and measurement of distancesand orientation of teeth with respect to each other.

[0009] In one aspect of the present invention, a method includesreceiving a plurality of two-dimensional images of an oral cavity, andgenerating at least one three-dimensional image of the oral cavity fromthe plurality of two-dimensional images. In another aspect of thepresent invention, a computerized apparatus includes a digitizerproviding five degrees of freedom, the digitizer having an arm, a chargecoupled device camera, rigidly mounted on the arm of the digitizer, anda computer, operably coupled to the digitizer and the camera thatreceives coordinate measurements from the digitizer and a plurality oftwo-dimensional images from the camera and generates a digitalthree-dimensional model from the coordinate measurements and from theplurality of two-dimensional images.

[0010] The present invention discloses the generation of a 3-D model ofthe jaw, not from a cast, but from the actual human jaw. The presentinvention disclose systems and method of data acquisition performeddirectly on a jaw using a small off the shelf charge coupled devicecamera in which the acquisition time is relatively short and is lessdiscomforting to the patient compared to current practices. The acquireddigital model is optionally stored with the patient data and optionallyretrieved on demand. The model optionally is transmitted over acommunication network to different remote practitioners for furtherassistance in diagnosis and treatment planning. Dental measurements andvirtual restoration are optionally performed and analyzed from thedigital model.

[0011] The present invention describes systems, clients, servers,methods, and computer-readable media of varying scope. In addition tothe aspects and advantages of the present invention described in thissummary, further aspects and advantages of the invention will becomeapparent by reference to the drawings and by reading the detaileddescription that follows.

BRIEF DESCRIPTION OF THE DRAWINGS

[0012]FIG. 1 is a block diagram of the hardware and operatingenvironment in which different embodiments of the invention can bepracticed.

[0013]FIG. 2 is a diagram illustrating a system-level overview of anembodiment of the invention.

[0014]FIG. 3 is a flowchart of a method for dental imaging, according toan embodiment of the invention.

[0015]FIG. 4 is a flowchart of a method for generating a 3-D image ofthe oral cavity from the plurality of 2-D images, according to anembodiment of the invention.

[0016]FIG. 5 is a flowchart of a method for generatingshape-from-shading data, according to an embodiment of the invention.

[0017]FIG. 6 is a diagram of surface triangulation, according to anembodiment of the invention.

[0018]FIG. 7 is a flowchart of a method for fusing the range data to theshape-from-shading data, according to an embodiment of the invention.

[0019]FIG. 8 is a block diagram of an apparatus for generating athree-dimensional digital image of a jaw, according to an embodiment ofthe invention.

[0020]FIG. 9 is a block diagram of an apparatus involved in calibratinga charge coupled device camera, according to an embodiment of theinvention.

DETAILED DESCRIPTION OF THE INVENTION

[0021] In the following detailed description of embodiments of theinvention, reference is made to the accompanying drawings that form apart hereof, and in which is shown by way of illustration specificembodiments in which the invention may be practiced. These embodimentsare described in sufficient detail to enable those skilled in the art topractice the invention, and it is to be understood that otherembodiments may be utilized and that logical, mechanical, electrical andother changes may be made without departing from the scope of thepresent invention. The following detailed description is, therefore, notto be taken in a limiting sense, and the scope of the present inventionis defined only by the appended claims.

[0022] The detailed description is divided into five sections. In thefirst section, the hardware and the operating environment in conjunctionwith which embodiments of the invention may be practiced are described.In the second section, a system level overview of the invention ispresented. In the third section, methods for an embodiment of theinvention are provided. In the fourth section, a particularobject-oriented Internet-based implementation of the invention isdescribed. Finally, in the fifth section, a conclusion of the detaileddescription is provided.

Hardware and Operating Environment

[0023]FIG. 1 is a block diagram of the hardware and operatingenvironment 100 in which different embodiments of the invention can bepracticed. The description of FIG. 1 provides an overview of computerhardware and a suitable computing environment in conjunction with whichsome embodiments of the present invention can be implemented.Embodiments of the present invention are described in terms of acomputer executing computer-executable instructions. However, someembodiments of the present invention can be implemented entirely incomputer hardware in which the computer-executable instructions areimplemented in read-only memory. One embodiment of the invention canalso be implemented in client/server computing environments where remotedevices that are linked through a communications network perform tasks.Program modules can be located in both local and remote memory storagedevices in a distributed computing environment.

[0024] Computer 110 is operatively coupled to display device 112,pointing device 115, and keyboard 116. Computer 110 includes a processor118, commercially available from Intel®, Motorola®, Cyrix® and others,random-access memory (RAM) 120, read-only memory (ROM) 122, and one ormore mass storage devices 124, and a system bus 126, that operativelycouples various system components including the system memory to theprocessing unit 118. Mass storage devices 124 are more specificallytypes of nonvolatile storage media and can include a hard disk drive, afloppy disk drive, an optical disk drive, and a tape cartridge drive.The memory 120, 122, and mass storage devices, 124, are types ofcomputer-readable media. A user enters commands and information into thecomputer 110 through input devices such as a pointing device 115 and akeyboard 116. Other input devices (not shown) can include a microphone,joystick, game pad, satellite dish, scanner, or the like. The processor118 executes computer programs stored on the computer-readable media.Embodiments of the present invention are not limited to any type ofcomputer 110. In varying embodiments, computer 110 comprises aPC-compatible computer, a MacOS®-compatible computer or aUNIX®-compatible computer. The construction and operation of suchcomputers are well known within the art.

[0025] Furthermore, computer 110 can be communicatively connected to theInternet 130 via a communication device 128. Internet 130 connectivityis well known within the art. In one embodiment, a communication device128 is a modem that responds to communication drivers to connect to theInternet via what is known in the art as a “dial-up connection.” Inanother embodiment, a communication device 128 is an Ethernet® orsimilar hardware (network) card connected to a local-area network (LAN)that itself is connected to the Internet via what is known in the art asa “direct connection” (e.g., T1 line, etc.).

[0026] Computer 110 can be operated using at least one operatingenvironment to provide a graphical user interface including auser-controllable pointer. Such operating environments include operatingsystems such as versions of the Microsoft Windows® and Apple MacOS®operating systems well-known in the art. Embodiments of the presentinvention are not limited to any particular operating environment,however, and the construction and use of such operating environments arewell known within the art. Computer 110 can have at least one webbrowser application program executing within at least one operatingenvironment, to permit users of computer 110 to access intranet orInternet world-wide-web pages as addressed by Universal Resource Locator(URL) addresses. Such browser application programs include NetscapeNavigator® and Microsoft Internet Explorer®.

[0027] Display device 112 permits the display of information, includingcomputer, video and other information, for viewing by a user of thecomputer. Embodiments of the present invention are not limited to anyparticular display device 112. Such display devices include cathode raytube (CRT) displays (monitors), as well as flat panel displays such asliquid crystal displays (LCD's). Display device 112 is connected to thesystem bus 126. In addition to a monitor, computers typically includeother peripheral input/output devices such as printers (not shown),speakers, pointing devices and a keyboard. Speakers 113 and 114 enablethe audio output of signals. Speakers 113 and 114 are also connected tothe system bus 126. Pointing device 115 permits the control of thescreen pointer provided by the graphical user interface (GUI) ofoperating systems such as versions of Microsoft Windows®. Embodiments ofthe present invention are not limited to any particular pointing device115. Such pointing devices include mouses, touch pads, trackballs,remote controls and point sticks. Finally, keyboard 116 permits entry oftextual information into computer 110, as known within the art, andembodiments of the present invention are not limited to any particulartype of keyboard.

[0028] The computer 110 can operate in a networked environment usinglogical connections to one or more remote computers, such as remotecomputer 150. The logical connections are achieved by a communicationdevice coupled to, or a part of, the computer 110. Embodiments of thepresent invention are not limited to a particular type of communicationsdevice. The remote computer 150 can be another computer, a server, arouter, a network PC, a client, a peer device or other common networknode. The logical connections depicted in FIG. 1 include a local-areanetwork (LAN) 151 and a wide-area network (WAN) 152. Such networkingenvironments are commonplace in offices, enterprise-wide computernetworks, intranets and the Internet.

[0029] When used in a LAN-networking environment, the computer 110 andremote computer 150 are connected to the local network 151 through anetwork interface or adapter 153, which is one type of communicationsdevice. When used in a conventional WAN-networking environment, thecomputer 110 and remote computer 150 communicate with a WAN 152 throughmodems (not shown). The modem, which can be internal or external, isconnected to the system bus 126. In a networked environment, programmodules depicted relative to the computer 110, or portions thereof, canbe stored in the remote memory storage device.

System Level Overview

[0030]FIG. 2 is a block diagram that provides a system level overview ofthe operation of embodiments of the present invention. Embodiments ofthe invention are described as operating in a multi-processing,multi-threaded operating environment on a computer, such as computer 110in FIG. 1.

[0031] System 200 includes a small intra-oral charge coupled device(CCD) camera 210 to enable the first stage of data acquisition 220. Inone embodiment, the camera 210 includes a built-in white light, rigidlymounted on a five-link 3-D digitizer arm. The camera 210 is calibratedand then placed inside the oral cavity. The camera 210 acquires a set ofoverlapping images 230 {I_(j)|j=1,2, . . . , J} for various portions ofthe jaw such that U_(j=1)I_(j) covers the entire jaw. The images 230 arepreprocessed to reduce noise, sharpen edges, and remove specularity 240.Removing specularity 240 is improves the accuracy of the reconstructedsurfaces using a shape-from-shading (SFS) process 250. In one embodimentof removing specularity, changes in the reflection map are removed froma luminance image by calculating the logarithmic gradient of the image230 and thresholding at locations of abrupt chromaticity change. Inanother embodiment, median filtering is implemented to remove specklenoise from the images. Using a SFS process 250 that accounts for thecamera perspective projection 255, J sets of 3-D points are computed. Toobtain accurate metric measurements, range data 262 is obtained usingthe five-link digitizer 260. The range data comprises reference pointson the jaw. Fusion 265 of the range data and the SFS output providesaccurate metric information that can be used later for orthodonticmeasurements and implant planning. System 200 also includes aregistration 270 technique that merges the resulting 3-D points toobtain a complete 3-D description of the jaw 275. The 3-D description275 is transformed into patches of free form surfaces using atriangulation process 280. The triangulation process 280 enablesoptional development of a 3-D solid model 285 for visualization.Optionally, a cast 290 is fabricated from the 3-D description 275 viarapid prototyping. Further optional processing of the digital model 275includes tooth separation 295, force analysis, implant planning, andsurgical simulation 298.

[0032] The system level overview of the operation of an embodiment ofthe invention has been described in this section of the detaileddescription. While the invention is not limited to any particular camera210, image acquisition system 220, data fusion 265 and registrationprocess 270, triangulation process 280, and cast 290, for sake ofclarity a simplified camera 210, image acquisition system 220, datafusion 265 and registration process 270, triangulation process 280, andcast 290, has been described.

Methods of an Embodiment of the Invention

[0033] In the previous section, a system level overview of the operationof an embodiment of the invention was described. In this section, theparticular methods of such an embodiment are described by reference to aseries of flowcharts. Describing the methods by reference to a flowchartenables one skilled in the art to develop such programs, firmware, orhardware, including such instructions to carry out the methods onsuitable computers (the processor of the computers executing theinstructions from computer-readable media). Similarly, the methodsperformed by the server computer programs, firmware, or hardware arealso composed of computer-executable instructions. Methods 300-500 and700 are performed by a program executing on, or performed by firmware orhardware that is a part of, a computer, such as computer 110 in FIG. 1

[0034]FIG. 3 is a flowchart of a method 300 for dental imaging,according to an embodiment of the invention.

[0035] Method 300 includes receiving 310 a plurality of two-dimensional(2-D) images of an oral cavity. In varying embodiments, the oral cavityis a mammalian oral cavity and/or a human oral cavity. In otherembodiments, the plurality of 2-D images are a plurality of 2-D opticalimages.

[0036] Thereafter, method 300 includes generating 320 at least onethree-dimensional (3-D) image of the oral cavity from the plurality of2-D images.

[0037] Optionally, method 300 also includes constructing 330 a physicalcast of the oral cavity from the 3-D image. In varying embodiments, thephysical cast is a plastic cast formed by a rapid prototyping machine,or a plaster cast.

[0038] Optionally, method 300 includes storing 340 the 3-D image inassociation with patient information. For example, the 3-D image isstored in a database of patient records.

[0039] Optionally, method 300 further includes modifying 350 the 3-Dimage in accordance with a proposed or suggested orthodontic or dentaltreatment of the patient.

[0040] In another embodiment of method 300, the 2-D images that arereceived in action 310, are beforehand generated from a common referencepoint in 3-D space.

[0041]FIG. 4 is a flowchart of a method 400 for generating a 3-D imageof the oral cavity from the plurality of 2-D images, according to anembodiment of the invention.

[0042] Method 400 includes generating 410 shape-from-shading (SFS) datafrom the plurality of 2-D images using a shape-from-shading process, theshape-from-shading data comprising a first plurality of 3-D points.Generating SFS data is disclosed in further detail in method 500 in FIG.5.

[0043] Method 400 also includes generating 420 range data comprising asecond plurality of 3-D points from the plurality of 2-D images using arange-data process.

[0044] Method 400 further includes fusing 430 the range data to theshape-from-shading data, yielding fused data comprising a thirdplurality of 3-D points. Fusing 430 is disclosed in further detail inmethod 700 in FIG. 7.

[0045] Thereafter, method 400 includes registering 440 the fused data,yielding registered data comprising a fourth plurality of 3-D points.

[0046] Subsequently, method 400 includes triangulating 450 theregistered data, yielding the at least one 3-D image of the oral cavity.

[0047]FIG. 5 is a flowchart of a method 500 for generatingshape-from-shading data, according to an embodiment of the invention.

[0048] SFS assumes that the surface orientation at a point M on asurface S is determined by the unit vector perpendicular to the planetangent to S at M. Under the assumption of orthographic projections, theelemental change in the depth Z at an image point (x,y) can be expressedas${\delta \quad z} \approx {{\frac{\partial Z}{\partial x}\delta \quad x} + {\frac{\partial Z}{\partial y}\delta \quad {y.}}}$

[0049] The partial derivatives are called surface gradients (p,q). Thenormal to a surface patch is related to the surface gradient byn=(p,q,1). Assuming that surface patches are homogeneous and uniformlylit by distant light sources, the brightness E(x,y) seen at the imageplane often depends only on the orientation of the surface. Thisdependence of brightness on surface orientation can be represented as afunction R(·) defined on the Gaussian sphere. Thus, the SFS problem isformulated as finding a solution to the brightness equation:E(x,y)=R(p,q,L), where R(p,q,L) is the surface reflectance map and L isthe illuminant direction.

[0050] Method 500 includes estimating 510 the direction of theilluminant from the plurality of 2-D images, in reference to cameraintrinsic parameters.

[0051] In the present invention, a white light beam is built in the CCDcamera, yielding a valid assumption that the illuminant direction isknown. However, an assumption of orthographic projection is not adequatefor the dental application because the camera is very close to theobject. Conventional SFS methods using perspective projection ignore thecamera extrinsic parameters, hence cannot provide metric information ofthe depth. In the present invention, the CCD camera is calibrated andthe camera parameters are used in the SFS method to obtain a metricrepresentation of the teeth and gum surfaces. To calibrate the camera,the relation between the 3D point M={X,Y,Z} and the corresponding imagecoordinates m={x,y} is written as; {right arrow over (sm)}={right arrowover (PM)} where s is a scalar, {right arrow over (m)} and {right arrowover (M)} are the extended vectors [m^(T)1]^(T) and [M^(T)1]^(T), and Pis called the camera calibration matrix. In general, P=A [R,t] where Ais a matrix containing all the camera intrinsic parameters and R,t arethe rotation matrix and translation vector. The matrix P has 12 elementsbut has only 11 degrees of freedom because it is defined up to a scalefactor.

[0052] The standard method of calibration is to use an object with knownsize and shape and extract the reference points from the object image.It can be shown that given N points (N>=6) in general positions, thecamera can be calibrated. The perspective projection matrix P can bedecomposed as [Bb] where B is a 3×3 matrix and b is a 3×1 vector suchthat: TABLE 1 s{right arrow over (m)} = BM + b

[0053] or, TABLE 2 M = B⁻¹ (s{right arrow over (m)} − b) = f(s(x,y))

[0054] This last equation represents a line in the 3D spacecorresponding to the visual ray passing through the optical center andthe projected point m. By finding the scalar s, ƒ(s(x,y)) will define aunique 3D point M on the object. The surface normal at M is defined tobe the cross product of the two gradient vectors${p = \frac{{f\left( {s\left( {x,y} \right)} \right)}}{x}},{q = {\frac{{f\left( {s\left( {x,y} \right)} \right)}}{y}.}}$

[0055] The surface reflectance R(,) becomes a function of the scalarsdefined in The equation in table 1, i.e., TABLE 3${R(s)} = \frac{\left( {p \times q} \right) \cdot L}{{{p \times q}}\quad {L}}$

[0056] Method 500 also includes determining 520 a solution to abrightness equation from the direction of the illuminant, yielding theshape-from-shading data comprising a first plurality of 3-D points.

[0057] The formulation of the SFS problem becomes finding the scalar sthat solves the brightness equation g(s)=E(x,y)□R(s)=0. In oneembodiment, the brightness equation is solved using a Taylor's seriesexpansion and applying a Jacoby iterative method [ ]. After niterations, for each point (x,y) in the image, s_(x,y) ^(n) is given asfollows TABLE 4$s_{x,y}^{n} = {s_{x,y}^{n - 1} + \frac{- {g\left( s_{x,y}^{n - 1} \right)}}{\frac{}{s_{x,y}}{g\left( s_{x,y}^{n - 1} \right)}}}$

[0058] where, TABLE 5${\frac{}{s_{x,y}}{g\left( s_{x,y} \right)}} = {{- \frac{N}{s_{x,y}}} \cdot \frac{L}{L}}$

[0059] TABLE 6$\frac{N}{s_{x,y}} = {{\frac{v}{s_{x,y}}\quad \frac{1}{\sqrt{v^{t}v}}} - {\frac{v}{\sqrt{\left( {v^{t}v} \right)^{3}}}\left( {v^{t}\frac{v}{s_{x,y}}} \right)}}$

[0060] TABLE 7$\frac{v}{s_{x,y}} = {{B^{- 1}\overset{\rightarrow}{m} \times {B^{- 1}\left( {0,s_{x,{y - 1}},0} \right)}^{t}} + {{B^{- 1}\left( {{s_{{x - 1},y,}0},0} \right)}^{t} \times B^{- 1}\overset{\rightarrow}{m}}}$

where v = p × q.

[0061]FIG. 6 is a diagram 600 of surface triangulation, according to anembodiment of the invention.

[0062] Diagram 600 includes the image plane 610 as a set of triangularpatches each with a normal N 620, the dot product of the light directionL 630 and the normal N 620 is called the reflectance which determinesthe image intensity at the corresponding pixel location m 640. Imageplane 610 is described as having 2-D coordinates x 650 and y 660. Whenpixel location m 640 is projected beyond optical center 670, in lightdirection L 630, the projection yields form M 680 having points M₁ 685and M₂ 687. The diagram is also described in relation to 3-D coordinatesX 690, Y 693, and Z 686.

[0063] In another embodiment of method 500, method 500 is described interms of the FIG. 6 by: TABLE 8$M = {B^{- 1}\left( {{s_{x,y}\begin{pmatrix}x \\y \\1\end{pmatrix}} - b} \right)}$

$M_{1} = {B^{- 1}\left( {{s_{{x - 1},y}\begin{pmatrix}{x - 1} \\y \\1\end{pmatrix}} - b} \right)}$

$M_{2} = {B^{- 1}\left( {{s_{x,{y - 1}}\begin{pmatrix}x \\{y - 1} \\1\end{pmatrix}} - b} \right)}$

p = M − M₁

[0064] TABLE 9$= {{{B^{- 1}\left( {s_{x,y} - s_{{x - 1},y}} \right)}\begin{pmatrix}x \\y \\1\end{pmatrix}} + {B^{- 1}s_{x - 1}{y\begin{pmatrix}1 \\0 \\0\end{pmatrix}}}}$

q = M − M₂

[0065] TABLE 10$= {{{B^{- 1}\left( {s_{x,y} - s_{x,{y - 1}}} \right)}\begin{pmatrix}x \\y \\1\end{pmatrix}} + {B^{- 1}{s_{x,{y - 1}}\begin{pmatrix}1 \\0 \\0\end{pmatrix}}}}$

[0066] A unit normal N to the patch formed by M, M₁, and M₂ iscalculated as follows: TABLE 11 $N = \frac{p \times q}{{p \times q}}$

[0067] A reflection function defined by the SFS is: TABLE 12${R( \cdot )} = \frac{N \cdot L}{L}$

[0068] Thus, the SFS brightness equation becomes: TABLE 13${g\left( s_{x,y} \right)} = {{E\left( {x,y} \right)} - \frac{N \cdot L}{L}}$

[0069] The solution to the SFS problem is to find s_(x,y) such that g(.)is minimized. Using Taylor expansion and Jacoby iterative methods,s(x,y) can be found by iteration as follows: TABLE 13$s_{s,y}^{n} = {s_{x,y}^{n - 1} + \frac{- {g\left( s_{x,y}^{\text{n} - 1} \right)}}{\frac{}{s_{m,y}}{g\left( s_{x,y}^{n - 1} \right)}}}$

[0070] TABLE 15${\frac{}{s_{x,y}}{g\left( s_{x,y} \right)}} = {\frac{N}{s_{x,y}} \cdot \frac{L}{L}}$

[0071] TABLE 16${{Let}\quad N} = {{\frac{v(s)}{\sqrt{{v(s)}^{t}{v(s)}}}\quad {where}\quad {v(s)}} = {p \times q}}$

[0072] TABLE 17$\frac{N}{s_{x,y}} = {{\frac{v}{s_{x,y}}\quad \frac{1}{\sqrt{v^{t}v}}} - {\frac{v}{\sqrt{\left( {v^{t}v} \right)^{3}}}\left( {v^{t}\frac{v}{s_{x,y}}} \right)}}$

$\begin{matrix}{\frac{v}{s_{x,y}} = {\frac{}{s_{x,y}}\left( {p \times q} \right)}} \\{= {{\frac{p}{s_{x,y}} \times q} + {p \times \frac{q}{s_{x,y}}}}}\end{matrix}\quad$

$\frac{p}{s_{x,y}} = {\frac{q}{s_{x,y}} = {B^{- 1}\begin{pmatrix}x \\y \\1\end{pmatrix}}}$

$\frac{v}{s_{x,y}} = {{{B^{- 1}\begin{pmatrix}x \\y \\1\end{pmatrix}} \times {B^{- 1}\begin{pmatrix}0 \\{s_{x,y} - 1} \\0\end{pmatrix}}} + {{B^{- 1}\begin{pmatrix}s_{{x - 1},y} \\0 \\0\end{pmatrix}} \times {B^{- 1}\begin{pmatrix}x \\y \\1\end{pmatrix}}}}$

[0073] The actions involved in the implementation include reading imageE(x,y), light direction L, and camera parameters (B,b). Initializings_(x,y)=0.01. □∀x,y get p and q as shown in the equations in tables 9and 10. Calculating N as shown in equation in table 11. Obtaining theerror using the brightness equation in table 13. Estimating the news_(x,y) using the equations in tables 14, 15, 16, and 17. Repeating theactions of ∀, calculating and estimating, until max_(x,y)|g(S_(x,y))|<εwhere ε is a predefined positive threshold. Thereafter, recovering thesurface 3-D points using equation in table 8, and constructingtriangular patches as shown in FIG. 6.

[0074]FIG. 7 is a flowchart of a method 700 for fusing the range data tothe shape-from-shading data, as in the fusing action 430 in FIG. 4,according to an embodiment of the invention.

[0075] The most important information for reconstructing an accurate 3-Dvisible surface, which is missing in shape from shading, is the metricmeasurement. Shape from shading also suffers from the discontinuitiesdue to highly textured surfaces and different albedo. The integration ofthe dense depth map obtained from SFS with sparse depth measurementsobtained from a coordinate measurement machine (CMM) for thereconstruction of 3-D surfaces with accurate metric measurements has twoadvantages. First, the integration removes ambiguity of the 3-D visiblesurface discontinuities produced by shape from shading. Second, theintegration complements missing metric information in the shape fromshading. The integration process, as depicted in FIG. 7, includes thefollowing stages. First, calculating 710 the error difference in theavailable depth measurements between the two sets of sensory data.Thereafter, approximating 720 a surface that fits this error difference.Subsequently, correcting 730 the shape from shading.

[0076] Method 700 includes calculating 710 the error difference inavailable depth measurements of the range data and theshape-from-shading data.

[0077] A multi-layer neural network is used for the surfaceapproximation process since neural networks was shown to be more robustin terms of complexity, speed and accuracy than other computationalapproaches (e.g., regularization techniques). The learning algorithmapplied is an extended Kalman-filter learning technique because of fastcomputation of weights generated by Kalman-filter. The x- andy-coordinates of the data points are the input to the network, while theerror in the depth value at the point (x,y) is the desired response. Theerror difference between the SFS and the range measurements and theirx-y coordinates are used to form the training set. The input to thenetwork is the x-y coordinates and the output is the error difference atthat coordinate.

[0078] Method 700 includes approximating 720 a surface the fits theerror difference, yielding an approximated surface.

[0079] Once training is accomplished, the neural network provides theapproximated smooth surface that contains information about the errorsin the shape from shading at the locations with no range data.

[0080] Method 700 includes correcting 730 the shape-from-shading datafrom the approximated surface, yielding fused data comprising a thirdplurality of 3-D points.

[0081] This approximated surface is then added to the SFS. The result isthe 3-D surface reconstruction that contains accurate metric informationabout the visible surface of the sensed 3-D object. The output of thefusion algorithm to each image is a set of 3-D points describing theteeth surfaces in this segment. To compensate for some digitizerinaccuracy in determining the camera location in space and theoccasional patient movement, The present invention in some embodimentsincludes a 3-D registration technique to link the 3-D points of all thesegments to produce one set of 3-D points describing the whole jawsurface.

[0082] In one embodiment, methods 300-500 and 700 are implemented as acomputer data signal embodied in a carrier wave, that represents asequence of instructions which, when executed by a processor, such asprocessor 118 in FIG. 1, cause the processor to perform the respectivemethod.

[0083] In another embodiment, methods 300-500 and 700 are implemented asa computer-readable medium having computer-executable instructions tocause a computer, such as computer 110, to perform the respectivemethod.

Apparatus Implementation

[0084] Referring to FIG. 8-9, a particular implementation of theinvention is described in conjunction with the hardware and operatingenvironment 100, the system overview in FIG. 2 and the methods describedin conjunction with FIGS. 3-5 and 7.

[0085]FIG. 8 is a block diagram of an apparatus 800 for generating athree-dimensional digital image of a jaw, according to an embodiment ofthe invention.

[0086] Apparatus 800 includes a ⅓ inch charge coupled device (CCD) colorcamera 810, such as camera 210 in FIG. 2. The camera 810 has 768H×494Veffective picture elements. In one embodiment, the camera also includesa 5.5 mm lens. Apparatus 800 also includes a 150 watt direct currentregulated white light source 820. Through a fiber optic bundle (notshown) that surrounds the CCD camera 810, the light source 820illuminates an oral cavity with a stable white light. The lightintensity is manually adjustable to control a shading effect. Apparatus800 also includes a computer 830, such as computer 110 in FIG. 1, thathosts the software required for the data processing, reconstruction andvisualization of the three-dimensional (3-D) jaw model, as described inmethods 300-500 and 700. Computer 830 also includes a 3-D digitizer 850,capable of digitizing a working space up to 1.27 m (sphere) with asampling rate of 1000 points/second. 3-D digitizer 850 is also shown asfive-link digitizer 260 in FIG. 2.

[0087] The CCD camera 810 is mounted on a stylus 840 of the 3-Ddigitizer 850 and its focal distance 870 is adjusted such that the imagewill be in focus only when the stylus tip touches a tooth surface. Animage plane 860 is normal to the stylus and the stylus tip is at pixel(0,0). Apparatus 800 also includes a display 880, such as display 112 inFIG. 1 to display the 3-D jaw model

[0088] The CCD camera 810 is calibrated as shown in FIG. 9.

[0089]FIG. 9 is a block diagram of an apparatus 900 involved incalibrating a charge coupled device camera, according to an embodimentof the invention. A coordinates measuring system is used to find thetransformation matrix T_(oc) 950 between the optical center M_(oc) ofthe camera at the calibration time 920 and the new location 930 whileacquiring the images. This transformation matrix is used to obtain thenew camera perspective projection matrix.

[0090] Camera 910 calibration is performed once before using the camera910. Where the camera 910 is stationary, the camera 910 does not need tobe re-calibrated again. Yet in some embodiments, the camera 910 will bemoving. Camera 910 movement requires the recalculation of theperspective projection matrix. However, as the camera 910 is mounted ona coordinates measuring system, the location of the optical centerM_(oc) can be tracked as the camera 910 moves from original opticalcenter M_(oc) 930 to new optical center M_(oc) 930, and the camera 910perspective projection can be recalculated.

[0091] The five degrees of freedom provided by an arm 940 enables theacquisition of a sequence of intra-oral images covering the upper/lowerjaw. Also, with each image, the camera 910 location in the 3-D space ismeasured. The perspective projection matrix is re-adjusted and thelocation and direction of the first pixel in the image are included.This information is used in the data fusion and registration phase toreference the image plane in the workspace.

[0092] In a computer-readable program embodiment, programs implementingmethods 300-500 and 700 can be structured in an object-orientation usingan object-oriented language such as Java, Smalltalk or C++, and theprograms can be structured in a procedural-orientation using aprocedural language such as COBOL or C. The software componentscommunicate in any of a number of means that are well-known to thoseskilled in the art, such as application program interfaces (A.P.I.) orinterprocess communication techniques such as remote procedure call(R.P.C.), common object request broker architecture (CORBA), ComponentObject Model (COM), Distributed Component Object Model (DCOM),Distributed System Object Model (DSOM) and Remote Method Invocation(RMI). The components execute on as few as one computer as in computer110 in FIG. 1, or on at least as many computers as there are components.

[0093] Reconstructing the 3-D model of the human jaw includes bycapturing a sequence of video images using a small intra-oral CCDcamera. These images are preprocessed to remove specularity. Referencepoints are obtained using the coordinate measurement machine (CMM)system. The range data are fused to the shape from shading (SFS) outputand then registration takes place. A cloud of points representing thejaw is obtained and, by triangulation, a solid digital model is formed.This model is optionally reproduced using a rapid prototype machine.Further analysis and orthodontics application can be performed on thedigital model.

Conclusion

[0094] Systems and methods of a model-based vision system for dentistrythat assists in diagnosis, treatment planning and surgical simulationhave been described. Although specific embodiments have been illustratedand described herein, it will be appreciated by those of ordinary skillin the art that any arrangement which is calculated to achieve the samepurpose may be substituted for the specific embodiments shown. Thisapplication is intended to cover any adaptations or variations of thepresent invention. One of ordinary skill in the art will appreciate thatthe invention can be implemented in a procedural design architecture, anobject-oriented architecture, or any other design architecture thatprovides the required relationships.

[0095] The present invention includes an integrated computer visionsystem that constructs a three-dimensional (3-D) model of the patient'sdental occlusion using an intra-oral video camera. A modified shape fromshading (SFS) technique, using perspective projection and cameracalibration, extracts the 3-D information from a sequence oftwo-dimensional (2-D) images of the jaw. Data fusion of range data and3-D registration techniques develop the complete jaw model.Triangulation is then performed, and a solid 3-D model is reconstructed.

[0096] In particular, one of skill in the art will readily appreciatethat the names of the methods and apparatus are not intended to limitembodiments of the invention. Furthermore, additional methods andapparatus can be added to the components, functions can be rearrangedamong the components, and new components to correspond to futureenhancements and physical devices used in embodiments of the inventioncan be introduced without departing from the scope of embodiments of theinvention. One of skill in the art will readily recognize thatembodiments of the invention are applicable to future communicationdevices, different file systems, and new data types.

[0097] The terminology used in this application with respect to is meantto include all object-oriented, database and communication environmentsand alternate technologies which provide the same functionality asdescribed herein. Therefore, it is manifestly intended that thisinvention be limited only by the following claims and equivalentsthereof.

We claim:
 1. A computerized method for dental imaging comprising:receiving a plurality of two-dimensional images of a oral cavity; andgenerating at least one three-dimensional image of the oral cavity fromthe plurality of two-dimensional images.
 2. The computerized method ofclaim 1, wherein the plurality of two-dimensional images furthercomprises a plurality of two-dimensional optical images.
 3. Thecomputerized method of claim 1, further comprising: constructing aphysical cast of the oral cavity from the three-dimensional image. 4.The computerized method of claim 1, further comprising: generating theplurality of two-dimensional images of the oral cavity from a commonreference point in three-dimensional space.
 5. The computerized methodof claim 1, wherein the generating further comprises: generatingshape-from-shading data from the plurality of two-dimensional imagesusing a shape-from-shading process, the shape-from-shading datacomprising a first plurality of three-dimensional points; generatingrange data comprising a second plurality of three-dimensional pointsfrom the plurality of two-dimensional images using a range-data process;fusing the range data to the shape-from-shading data, yielding fuseddata comprising a third plurality of three-dimensional points;registering the fused data, yielding registered data comprising a fourthplurality of three-dimensional points; and triangulating the registereddata, yielding the at least one three-dimensional image of the oralcavity.
 6. The computerized method of claim 5, wherein the generatingshape-from-shading data further comprises: estimating the direction ofthe illuminant from the plurality of two-dimensional images, inreference to camera intrinsic parameters; and determining a solution toa brightness equation, yielding the shape-from-shading data comprising afirst plurality of three-dimensional points.
 7. The computerized methodof claim 5, wherein the fusing the range data to the shape-from-shadingdata further comprises: calculating the error difference in availabledepth measurements of the range data and the shape-from-shading data;approximating a surface the fits the error difference, yielding anapproximated surface; and correcting the shape-from-shading data fromthe approximated surface, yielding fused data comprising a thirdplurality of three-dimensional points;
 8. A computer-readable mediumhaving computer-executable instructions to cause a computer to perform amethod comprising: receiving a plurality of two-dimensional opticalimages of an oral cavity; and generating at least one three-dimensionalimage of the oral cavity from the plurality of two-dimensional images.9. The computerized method of claim 8, further comprising: constructinga physical cast of the oral cavity from the three-dimensional image. 10.The computerized method of claim 8, further comprising: generating theplurality of two-dimensional images of the oral cavity from a commonreference point in three-dimensional space.
 11. The computerized methodof claim 8, wherein the generating further comprises: generatingshape-from-shading data from the plurality of two-dimensional imagesusing a shape-from-shading process, the shape-from-shading datacomprising a first plurality of three-dimensional points; generatingrange data comprising a second plurality of three-dimensional pointsfrom the plurality of two-dimensional images using a range-data process;fusing the range data to the shape-from-shading data, yielding fuseddata comprising a third plurality of three-dimensional points;registering the fused data, yielding registered data comprising a fourthplurality of three-dimensional points; and triangulating the registereddata, yielding the at least one three-dimensional image of the oralcavity.
 12. The computerized method of claim 11, wherein the generatingshape-from-shading data further comprises: estimating the direction ofthe illuminant from the plurality of two-dimensional images, inreference to camera intrinsic parameters; and determining a solution toa brightness equation, yielding the shape-from-shading data comprising afirst plurality of three-dimensional points.
 13. The computerized methodof claim 11, wherein the fusing the range data to the shape-from-shadingdata further comprises: calculating the error difference in availabledepth measurements of the range data and the shape-from-shading data;approximating a surface the fits the error difference, yielding anapproximated surface; and correcting the shape-from-shading data fromthe approximated surface, yielding fused data comprising a thirdplurality of three-dimensional points;
 14. A three-dimensional digitalimage of a human oral cavity produced by the process comprising:generating a plurality of two-dimensional optical images of the oralcavity from a common reference point in three-dimensional space;generating shape-from-shading data from the plurality of two-dimensionalimages using a shape-from-shading process, the shape-from-shading datacomprising a first plurality of three-dimensional points; generatingrange data comprising a second plurality of three-dimensional pointsfrom the plurality of two-dimensional images using a range-data process;fusing the range data to the shape-from-shading data, yielding fuseddata comprising a third plurality of three-dimensional points;registering the fused data, yielding registered data comprising a fourthplurality of three-dimensional points; and triangulating the registereddata, yielding the one three-dimensional image of the oral cavity. 15.The three-dimensional digital image of a human oral cavity of claim 14,produced by the process wherein generating shape-from-shading datafurther comprises: estimating the direction of the illuminant from theplurality of two-dimensional images, in reference to camera intrinsicparameters.
 16. A system for dental diagnosis comprising: a processor;and software means operative on the processor for generating athree-dimensional image of a human jaw, including generatingshape-from-shading data that is generated from a direction of anilluminant of the jaw that is estimated in reference to camera intrinsicparameters.
 17. A computerized system comprising: a digitizer providingfive degrees of freedom, having an arm; a charge coupled device camera,rigidly mounted on the arm of the digitizer; and a computer, operablycoupled to the digitizer and the camera; receiving coordinatemeasurements from the digitizer and a plurality of two-dimensionalimages from the camera; and generating a digital three-dimensional modelfrom the coordinate measurements and from the plurality oftwo-dimensional images.
 18. The computerized system of claim 17, furthercomprising: a rapid prototyping machine operably coupled to thecomputer, receiving the digital three-dimensional model and generating aphysical model of the digital three-dimensional model.
 19. Thecomputerized system of claim 17, further comprising: a display operablycoupled to the computer, receiving the digital three-dimensional modeland generating an image of the digital three-dimensional model.
 20. Thecomputerized system of claim 17, the computer further comprises: acomputer readable medium comprising means of: generatingshape-from-shading data from the plurality of two-dimensional imagesusing a shape-from-shading process, the shape-from-shading datacomprising a first plurality of three-dimensional points; generatingrange data comprising a second plurality of three-dimensional pointsfrom the plurality of two-dimensional images using a range-data process;fusing the range data to the shape-from-shading data, yielding fuseddata comprising a third plurality of three-dimensional points;registering the fused data, yielding registered data comprising a fourthplurality of three-dimensional points; and triangulating the registereddata, yielding the one three-dimensional image of the oral cavity.